Using ACL2 to Verify Security Properties of Specification- based Intrusion Detection Systems

Intrusion detection is considered to be an effective technique to detect attacks that violate the security policy of systems. There are basically three different kinds of intrusion detection: Anomaly detection, misuse detection and specification-based intrusion detection [MB02]. Specification-based intrusion detection differs from the others by describing the desired functionalities of security-critical entities including system programs, protocols, networks, and application programs [CK97]. This means unknown attacks will be detected as well as known attacks. There is an open question which kind of attacks can be detected by a specific specification-based intrusion detection system. In this paper a hierarchical model is built to reason specifications for different security requirements. A formal framework is built with ACL2 to analyze and improve detection rules of intrusion detection systems [KM00]. SHIM (System Health and Intrusion Monitoring) is used as an example to show the validation of our model and framework [CK01]. We formalize all specifications of SHIM and a trusted file policy and we reason about the soundness and completeness of the specifications by proving the specifications satisfy the policy with various assumptions. These assumptions are properties of the system that are not checked by the intrusion detection system. Analysis of these assumptions shows the role of SHIM in improving the security of the system.